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FOREWORD 

/ 

The Council on Postsecondary Accieflitation ^onsored a Summer 
Conference on Accreditation at the Kel'ogg Weit Center for Contin- 
uing Education at California State Polyt^f^-^hnic^/ University, Pomona, 
California, on August 2, 3, and 4» 1978 The cionference theme was 
"Evaluating Educational Quality"; ana about 150 accrediting profes- 
sionals^ commission members, and others jdined in the three-day 
discussion of this important and timely topic. / 

It quickly became apparent that there/ was strong agreement 
T among the conference participants that, irt evaluating educational 
quality, accrediting bodies increasingly must focus on "educational 
outcomes"— the product of education more than the process of the 
setting. This emphasis on educat»<^aai outcomes was shared in the 
major papers presented at the conference. They were: 

n A keynote address by Alexander W- Astin, Professor of Higher - 
Education, University of California, Los Angeles, and President, 
Higher Education Research Institute; 

D A discussion paper by Howard R. Bowen, Avery Professor of 
Economics and Education, Claremont Graduate School; and 

D A conference synthesis by Charles" M. Chambers, Staff Associ- : 
ate and Legal Advisor at COPA. 

These three papers so cogently and effectively address the topic 
of evaluating educational quality through an educational outcomes , 
approach that we are responding to the many requests that we pro- 
duce them in printed form. We hope that you share our belief that 
this publication mokes a valuable contribution to a growing literature 
on this significant topic. 

Kenneth E. Young 
President 

The Council on Postsecondary Accreditation 
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STUDENT-ORIENTED MANAGEMENT: 
A PROPOSAL FOR CHANGE 

Alexander W. Astin 

Two major studies that I have been involved in over the past 
several years have convinced me that our traditional notions about in- 
stitutional "quality" are in need of fundamental revision. 

These areas of research are student development and college 
administration. The substantial literature in these two areas shows 
very little overlap between the two. Studies on student development 
are carried out ' y a quite different group of researchers and read by 
a very different audience than studies on college management and 
administration. This gap probably is no accident; rather, it reflects 
something very fundamental about our institutions and about how 
they are perceived and operated. More important, the failure to ap- 
preciate the significance of this gap and to develop ways of bridging 
it represents a major obstacle to the improvement of institutions, 
whether by accrediting or any other means. 

My research convinces me that it is indeed possible to bridge the 
gap, but it will take considerable courage on the part of college ad- 
ministrators to take the first step. 

College Administration and Management 

Although most college catalogues claim that student development 
is a fundamental institutional purpose, the decision-making process 
in higher education typically ignores the student implications of 
alternative courses of action. This tendency is e^xemplified by th^ 
computer-based management information systems now used by 
many colleges and universities. Except for simplistic information on 
enrollments and majors, these systems provide almost no information 
- on students. Administrators who rely on such systems are thus en- 
couraged to view planning and decision-making basically as a problem 
in resource manipulation. The "benefit" side of the decision equation, 
as it reflects the probable consequences for stuc ont development, 
receives scant attention at best and in most cases is ignored altogether 
in the decision process. 

To the naive student development researcher, this problem has an 
obvious solution: **Let*s scrap these resource-oriented management 
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systems and develop instead systems that are primarily student- 
oriented/' However, administrators who mi^t be sympathetic to a 
more student-oriented approach are unlikely to support and pursue 
such a system simply because they believe it is unfeasible. I believe, 
on the other hand, that such a system not on)y is feasible but pro- 
vides an opportunity to improve substantially the quality of our 
planning and decision-making. 

Assumptions for the System 

The major assumptions upon which the approach would have to 
be based include: 

□ Assumption that the principal function of educational ad- 
ministration in general, and of planning and decision-making in 
particular, is to enhance student development. While the other major 
functions of higher education — research and public service — ^have 
considerable status in many institutions, administrators cannot hope 
to enhance the institution's educational mission if they ignore the 
student nplications of their decisions. 

□ Assumption that-the acquisition of resources — whether money, 
capital improvements, or additional staff — is regarded, at best, as an 
intermediate consequence of decisions; the ultimate consideration is 
how such resources can be utilized in the educational process to 
impact favorably on students. In other words, the acquisition of 
resources is not an end in itself - 

□ Assumption that student * 'development" is a multidimensional 
phenomenon involving cognitive skills, socialization, and career 
preparation. Developmental goals may be short-term (e.g., learning 
calculus or child psychology) or long-term (e.g., becoming a pro- 
ductive scientist or an effective parent). Although these different 
asp»ects of^ student development may be valued differently by stu- 
dents, parents, institutions, and the general public, the "quality** of 
an institution and the "effectiveness** of administration should be 
assessed in terms of the institution's contributions to these develop- 
mental goals. 

Theories About Administration and Learning 

One of the most important outcomes of our institute *s' studies of 
administration and of student development is that they have enabled 
us to develop some theories about how administrators operate and 
how students learn and develop. No aspect of the higher education 
enterprise — whether it be administration, accrediting, or assessing 
institutional quality — makes much sense unless it is based on some 
conception (theory oi: simply a point of view) about how administra- 
tors operate and how students develop. 



1. Piigher Education Research Institute. 



Most writings that deal ^ With the theory and practice of college 
administration say very little , about learning or education; some 
indicate that profit rather than education is the bottom line. Admin- 
istration is more understandable when viewed as an art form rather 
than a science that can be taught didactically or learned by reading a 
"how to." A manual on "how to administer a co'lege" is largely a 
waste of time in teaching people to be effective administrators. 

An analogy to the fine and performing arts is instructive. In these 
fields, an essential ingredient in the development of performing 
techniques or skills is feedback, or what the ler ning psychologists 
call "knowledge of results,*' Neophyte painters ee what comes out 
on the canvas and adjust their behavior accordingly; musicians hear 
what they play or sing; dancers use mirrors to enhance their feed- 
back. 

Administrators in other fields generally have feedback to gauge 
the effectiveness of their efforts— access to information on sales 
volume, profits and losses, etc.— to guide their planning and decision- 
making. The educational administrator, by contrast, ordinarily re- 
ceives little feedback on student learning and development to guide 
decisions. While college administrators typically have access to fiscal 
data, they have little information concerning the educational condi- 
tion of the institution; they lack regular feedback on the educational 
development and progress of their students. 

The same "need for feedback" argument could be made for col- 
lege faculty. Faculty are somewhat closer to appropriate feedback in 
their classroom teaching and advising experiences, but even there the 
feedback is often misleading, coming mostly from the more aggres- 
sive students. Further, such feedback may not tell the professor 
much about what students are actually learning. Relying on final 
examination results is not appropriate feedback about learning. 
for advising, professors rarely have an opportunity to learn about 
their successes and failures in this potentially important enterprise. In 
my own contacts with students, I seldom have a sense of what is really 
going on except among those few students with whom I work closely 
in a kind of apprenticeship relationship. 

The "performing art" analogy also can be extended to support 
staff. Many areas of institutional functioning impact directly on 
students: registration, orientation, financial aid, housing, food 
services, parking, social activities, career counseling, personal coun- 
seling, extracurricular activities, health services, and job placement. 
How can the personnel responsible for these diverse student services 
improve their programs and policies without systematic feedback 
concerning the impact of their efforts on the students they serve? 

Learning and Developing 

If a viable theory about sLudent development could be formu- 
lated, it should presumably enable administrators to choose among 
alternatives for solutions with a little more intelligence and a little 
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more consistency. My own theory of student development grew out 
of two major studies of institutional impact on student development.^ 
A key concept in the theory is "student involvement." The theoiy 
states that the more the student is involved in the academic experi- 
ence^ the greater the learning and growth. The less the involvement^ 
the less the learning and the greater the chance that the student will 
be dissatisfied and drop out. Involvement is not an esoteric or 
mysterious construct. It is manifest in how much physical and psy- 
chological energy the ^student devotes to the academic experience. In • 
certain respects, involvement very much resembles a Freudian concept 
called **cathexis." Freud had the idea that people invest psychologicsEil 
energy in things or people outside of themselves, i.e., persons can 
"cathect" on their friends, families, jobs, and so on. 

The students* degree of cathexis or involvement in the academic 
experience can be manifest in a variety of ways: the amount of time 
they spend studying, how much they interact with each other, or 
with professors, and the amount of time they spend on campus. 
There are, of course, other ways for estimating how students are 
involved in what is going on in the institution. 

Our longitudinal research shows that virtu;illy every institutional 
policy or practice that increases involvement also enhances student 
development. We found that, contrary to folklore, having the part- 
time job on campus actually helps the student to stay in college. 
(When a student works more than 20 hours, however, the effect dis- 
ap{>ears, and full-time work reverses the trend.) 

A second and perhaps the most important phenomenon that en- 
hances student learning and development is the residential exp>erience. 
We found also that being a member of a social fraternity or sorority 
enhances persistence. Participat^Lvn in moderation in almost any kind 
of extracurricular activity seems to help: honors programs, ROTC, 
undergraduate research projects, frequent interaction with faculty 
and with student peers, studying hard, athletics. 

Getting Students Involved 

Research results therefore suggest that, as a general operating 
principle, the administration and faculty should formulate plans and 
policies designed to encourage students to get more involved^ to 
invest more of their time and their physical and psychic energy in the 
educj-tional process. 

The student-oriented management information system would 
operate on a fairly simple feedback principle: Student progress would 
be monitored through an infe r motion system that regularly yields 
data on any aspect of personal development that relates to the insti- 
tution's educational goals. Subsequent decisions about institutional 



2. See A. W. Astin, Preventing Students from Dropping Out (San Francisco: 
Jossey-Bass, 1976), and Astin, Four Critical Years (San Francisco: Jossey-Bass, 
1977). 
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policy and practice, including resource allocation decisions, vouM be 
designed to bring about greater correspondence between student 
development and stated developmental goals. The system itself would 
be continuously subject to change through one or more of the fol- 
lowing mechanisms: 

n Redefinition of institutional objectives (addition of new objec- 
tives, sharpening definitions, and so on). 

O Modifications in the method to assess particular developmental 
outcomes. 

a Alterations in the method of analysis or dissemination of 
information. 

Design for Decision-Making 

Like any management information system, the student-oriented 
^stem should be designed primarily for use in decision-making. The 
need to render an educational decision implies the existence of two 
fundamental conditions: (1) some desired outcomes or ends, and 
(2) two or more alternative means to achieve those ends. The ends, 
of course, concern student development. Alternative means might 
include organizing certain learning experiences (e.g., curricula, in- 
structional methods), structuring the physical environment in the 
design and location of classrooms, buildings, open space, or establish- 
ing certain rules or regulations. Every administrative decision should 
be predicated on a belief in the existence of a causal *-elationship be- 
tween some educational outcome and a particular means selected to 
achieve that outcome. The good administrator should believe that of 
all the means available, the one selected is **best*' in that it is most 
likely to produce the desired outcome. 

lYaditdonally, college faculty and administrators have been more 
means- and ends-oriented. The reward structure in higher education 
clearly reinforces this tendency, since administrators are rewarded not 
for maximizing the development of the student, but for acquiring a 
large share of higher education's limited resources: money, bright 
students, and higrhly trained and prestigious faculty. The accreditation 
process reinforces ^his means-orentiation, because it seldom asks hard 
questions about ends. It is refreshing to see the accrediting bodies 
now placing more emphasis on having institutions and programs 
evaluate outcomes or ends. 

Another reason for the means-orientation of most administrators 
is that the causal connections between means and ends are not well 
understood. Consequently, administrators must operate with a largely 
untested folklore about what works and what doesn't work. Thus, 
one major benefit of the student-oriented system is that it would 
help administrators develop a better understanding of how their 
actions are likely to affect stadents. If nothing else, it would 
encoiirage them to think more in terms of ends than of means. 
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Ultimately, the system should be able to tell them if a particular 
program or policy actually worked ! 

Student Time as a Resource 

Administratoi.^i concentrating on resource acquisition often fail 
to recognize that their greatest potential resource may be student 
time. The theory of involvement suggests that the extent to which 
students are able to achieve particular developmental outcomes is a 
direct function of the amount of time and effort they devote to 
activities designed to produce these outcomes. For example, if 
improving knowledge of history is regarded as an important outcome 
for history majors, the maximization of this outcome may well be 
a direct function of the time the students spend listening to profes- 
sors talk about history, reading historical works, and discussing 
history Mth other students. Within certain broad limits, the more 
time spent in such activities, the greater the learning. The time spent 
attending formal lectures and taking notes and otherwise attem^>ting 
to comprehend the material represents a fraction of the potential 
time and effort that might be devoted to activities tha could con- 
tribute to historical knowledge and skills. The more obvious *nethods 
for controlling this time and effort would be to assign out-of-class 
work, to improve the quality or accessibility of library offerings in 
history, and to make lectures and other course materials available 
through audio and video tapes, slides, and other media. There is an 
important distinction, however, between making such resources avail- 
able and the student's effective use of them. A n institution may ha^ e 
an excellent library collection and a wealth of associated media thi t 
students seldom use. Professors may assign homework that students 
fail to complete. Classes may be poorly attended, or students may 
attend and fail to profit significantly because they are bored or dis- 
tracted. 

In short, an effectively managed institution not only provides 
appropriate learning resources but also creates the environmental 
conditions that encourage students to make effective use of those 
learning resources. 

Most administrators do not recognize that virtually every institu- 
tional policy and practice can affect how students spend their time 
and how much effort they devote in academic versus nonacademic 
pursuits. These "administrative" issues include class schedules, poli- 
cies on class attendance, regulations on academic probation and parti- 
cipation in honors courses, policies about office hours for faculty, 
and student orientation. Moreover, administrative decisions on many 
**non-academic" issues can significantly affect the way students spend 
their time in academic pursuits: location of new buildings, such as 
dormitories and student unions; rules l^oveming residency; design of 
recreational and living facilities; on-campus employment opportuni- 
ties; number and type of extracurricular activities and regulations 
governing participation; frequency, type, and cost of cultural events; 
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roommate assignments; Anancial aid policies; relative attractiveness of 
eating facilities on and off campus; and parking regulations. 

Measuring Student Development 

The success of any student^based management information sys- ' 
tern (MIS) depends heavily on the relevance of student outcome data 
to institutional objectives. While any given institution needs to d«*- 
velop its own specific outcome measures, at least three types of 
measures are relevant to the educational objectives of most institu- 
tions. These **core" measures should be included in any student-based 
management information system: 

O Successful completion of a program of study. In its simplest 
form, this measure would involve a dichotomy; the student either 
completes a degree plan or drops out. More sophisticated approaches 
to such a measure would be to determine if the students' undergrade 
uPte achievements are consistent with their original plans at entry. 
For example, if a student entered college to become a lawyer, simple 
completion of the undergraduate degree may not be sufficient, given 
the relatively stringent admissions requirements of law school. Thus, 
it might be more appropriate to use admission to law school as a 
criterion of "successful completion." Different students, in other 
words, may require different "success" criteria. 

D Cognitive development. Virtually all colleges and universities are 
concerned with students' cognitive development. Most instituitons, 
however, limit their assessment to the traditional grade-point average. 
Since grades reflect only students' relative level of performance at a 
particular point in time, they may not indicate accurately what the 
stuaent has learned. Cognitive development probably requires some 
form of repeated measurement, where change can be assessed by 
comparing the student performance level at two or more points in 
time. Research on learning, however, suggests that study time may be 
a good surrogate measure for how much the student is learning. 

p Student satisfaction. In one sense, the students' level of satis- 
faction with the institution's program is one of the most important 
indications of program effectiveness. Satisfaction can be measured in 
an overall sense, but it would probably be usefu. to obtain informa- 
tion on the students' degree of satisfaction with more specific mat- 
ters, such as the quality of teaching, advising, curriculum, facilities, 
extracurricular activities, and student services. 

Designing the Data Base 

The basic types of student data include entry, process, and out- 
come. Entry data refer to characteristics of students when they first 
enroll, process data to what hapi>ens to the students while enrolled, 
.and outcome data to the students' degree of attainment of desired 
educational or behavioral objectives. 
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The basic function of pro<?ess information (which encompasses 
much of the potential data that can be collected on students) is to 
reflect what is happening to students. The distinction between process 
and outcome information is often fuzzy. For example, the fact that a 
student switches majors can be treated as a change in process, since 
the student is now exposed to a different fit Id of study, or as an out- 
come, since the student usuaJly does the choosing. The distinction 
between process and outcome in this case is not intrinsic to the data; 
rather, it depends on how the data are Used. Thus, if one is interested 
in how the student's majojr field affects some other outcomes (e.g., 
the number of hours st:>ent studying) the change in major can be re- 
garded as a change in process. If one is interested in how the student*s 
choice of a major is affected by some other process variable, then the 
change in field is treated as an outcome variable. 

Student Satisfaction 

Student satisfaction is a useful category because it is relatively 
#»asy to assess and is widely applicable to the college experience. 
Students can evaluate not only their academic proprams and instruc- 
tion, but also institutional services: such as orientation; registration, 
financial aid^ academic advisement, career counseling, personal coun- 
seling, health services, job placement, and campus housing. Even 
residential facilities can be rated on privacy, roommate assignment, 
quietnc^^vS* food service, bathroom facilities, and programming (lec- 
tures, films). Students can evaluate extracurricular activities, oppor- 
tunities for independent study, social life, work experience, and con- 
tact with faculty members and student peers. While any institution 
must decide for itself which areas of satisfaction are most critical, 
student satisfaction represents a potentially rich source of outcome 
data for the MIS. 

Critical Issues in Implementation 

The top row of boxes in Figure 1 :;hows the five major steps in- 
volved in implementing a student-oriented management information 
system. The second row of boxes (broken lines) illustrates each 
general step with a specific example* In the first step (A), it is neces- 
sary to determine what items of information are most needed in the 
preliminary system. For example, a modest beginning might include 
a simple student survey thit includes a time diary together with 
ratings of various aspxects of the institutional experience (of which 
the quality of academic advising would be just one). Ideally, all seg- 
ments of the academic community would participate in designing the 
instrument, although it would be important to keep the survey short 
and simple in this early stage. 

The second step (B) involves the actual data collection, which 
would presumably be planned to occur at regular intervals (say, 
semi-annually). Two important considerations here are the sampling 
procedures and the method of data collection. Since one should be 
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Figure 1 

Basic Steps in Implementing a Student-Based Management Information System 
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able to create separate reports of results at the departmental or 
school level, sampling should insure that each relevant organizational 
sub-unit is represented by adequate numbers of students. In large 
institutions, samples of respondents could be rotated so that the 
same student would not be surveyed more than once a year. In 
smaller cc iieges, it might be necessary' to sample the same students 
more than once a year, particularly if separate breakdowns of results 
' by class (i.e., freshman, sophomore), sex, or other sub-groupings are 
desired. The po«sible survey methods are many — mail, personal inter- 
views, etc.— but perhaps the surest method is periodically to take a 
small amount of class time to administer the survey. 

Disseminating Survey Results 

Dissemination of survey results (Step C) is a critical step in the 
« ••: implementation process. I have already stressed the importance of 
''providing separate tabulations by department or sch.ool in addition to 
institution-wide "norms" based on all respondents. Such norms pro- 
vide a basis for each sub-unit to evaluate its own data. Ideally one 
would have access to comparative normative data from other institu- 
tions. 

Although such inter-institutional consortium arrangements may 
be diffi<nilt and time-consuming to implement, the resulting compara- 
tive norms can be very useful in providing a broader perspective from 
which to view any given institution's data. If several institutions are 
seriously considering a coopierative venture of this sort, it would not 
be necessary to use identical survey instruments (such a requirement 
might ultimately provide an insurmountable obstacle to implementa- 
tion). Perhaps the most realistic expectation is that some of the sur- 
vey items would be common to all institutions. 

Administrative Climate 

Another consideration in dissemination is the administrative 
climate in which the results are to be reported. If the administration 
sees its role as primarily educational— to help academic departments 
and administrative service units (financial aid, etc.) learn more about 
how they affect students— then the results are more likely to be 
viewed in, a constructive and facilitative way. On the other hand, if 
the administration intends to use the results as a basis for rewarding *' 
^ood i>erformance and punishing poor performance, then the entire 
survey activity is likely to be viewed with hostility and suspicion. 
Moreover, such an evaluative attitude will encoiiarge practitioners to 
manipulate the results. On a purely intuitive basis, I would predict 
that the survey's positive impact on educational practice will be 

' greatest when the professors and administrators -being rated believe 
that the results will not "be used to deprive them of resources or 

Q rewards. 

ERIC I* is difficult for a practicing professional to face up to his or her 
■■^limitatic^ns and to develop the courage to change established habits. 



If institutions real]y want to provide a more meaningful and produc- 
tive educational experience. for students, however, change must occur 
at the level of the individual teacher or administrator. These practi- 
tioners need to be convinced that they can be open to honest evalua- 
tion and criticism without fear of retaliation or rejection. Moreover, 
they need to believe that the administrative environment supports 
their attempts at self -evaluation and that constructive change is not 
only possible but actively supported by the administration. While 
such an atmosphere of openness clearly cannot be created overnight, 
the administration should strive to promote a positive and nonpuni- 
tive view of the student-based system from the ver> be^ning. 

Interpretation and Change 

The fourth implementation step (D) is the most difficult and 
complex one. It involves the interpretation of survey results and, 
where appropriate, the initiation of appropriate changes in policy or 
practice. In the hypothetical example shown in Figure 1, students in 
the social sciences have 'nted as "poor" the quality of academic ad- 
vising. Ideally one would hope that the social science departments 
would note this result, see it as a real problem, and undertake appro- 
priate remedial actions. It is possible, of-<rourse, that the relatively 
poor rating of advising in these fields would be interpreted by social 
science professors in other ways: the rating question is phrased in 
such a way as to prejudice the ratings by social science majors; social 
science majors are by nature more critical of their professors; and so 
, on. Sceptics should not jump to the conclusion that such interpreta- 
tions are merely defensive. Rather, the interpretations should be re- 
garded as testable propositions that can be examined by additional 
study: try out a new item to assess the effectiveness of advising; 
produce other evidence of the greater critical nature of social science 
mstfors: 

My hunch is that such intellectual issues would be actively 
.debated and tested only when an atmosphere of openness jand con- 
structive self -improvement is encouraged and rewarded. If top ad- 
ministrators must ultimately make hard choices about departmental 
competition for limited resources, the final decisions should favor 
and reward openness and receptivity to constructive cnange. 

The final -step (E) iii the implementation process completes the 
MIS cycle: The student body is resurveyed to assess change in institu- 
. tional effectiveness. Repeated measurement of this type not only 
provides an objective means for determining if remedial actions have 
been effective, but it also construes a kind of "early warning system" 
to detect unanticipated changes in institutional impact. It should be 
emphasized that the cyclical nature of the student-based information 
^stem satisfies^, two important needs: (1) to assess changes in institu- 
tional functioning; and- (2) to improve a mechanism for improving 
Pj^y„ student, information system by changing, deleting, or adding new 
Li^ns of data. •> % 



• ■ Figiii«2 

Sampli Output from a Student-Oriented MIS after Three Years: Quarterly Time Diary for Undergraduates 
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Student Time Studies 

/ Student time, of course, is a key category of process and outcome 
information. Some institutions may be reluctant to obtain time diaries 
i&om students because they assume the estimates will be crude and 
msensitive to changes in policy. A study of college environments 
several years ago indicated quite the contrary.^ Students attending a 
^national sample of 246 undergraduate institutions were asked how 
/hours per week, on the average, they spent in various activities. The 
/ variation among institutions was remarkable. These data showed 
clearly that student bodies can differ widely in the time they devote 
to activities- Regular monitoring is another means to assess the con- 
sequences of administrative decisions. 

Figure 2 contains a hypothetical chart showing how student time 
diary information can be utilized in institutional evaluation and de- 
cision-making. Time diary information is shown for three student 
behaviors: sleeping, studying, and commuting. In this hypothetical 
example, the institution has been collecting the information on a 
quarterly basis and has data from the last two academic years and 
from the first two quarters of the^ current academic year. Note that 
the average amount of time spent by the students sleeping has re- 
mained fairly stable at about 50 hours per week throughout the sur- 
vey period. Studying has shown a consistent pattern of declining 
hours during the spring of the past two years. However, during the 
first two quarters, bf the current year, study hours fail to show the 
increase observed in earlier years. This failure to replicate earlier 
trends suggests that other factors may have operated to reduce the 
number of hours that students devote to studying. In other words, 
the failure to replicate trends from earlier years may be an '*early 
warning sign" of potential problems. Hours spent commuting have 
been extremely consistent during the first two years'but have shown 
an increase during the current year. 

Although the data do not provide a basis for drawing firm con- 
clusions about the reasons for the decline in study hours, the increase 
in commuting time suggests a possible explanation: students may be 
studying less because they are commuting more! These concomitant 
trends thus provide a basis for institutional decision-makers to de- - 
velop and test alternative hypotheses to account for these trends. For 
example, it may be that changes in parking regulations introduced in 
the current year may-have increased the amount of commuting time. 
Or, changes in admissions and requirement policies may have increased - 
the number of commuters. To test the association between com- 
muting and study hours, institutional administrators could cross- 
tabulate commuting and studying, tp see if students who commute 
more study less. There may be other explanations, but the basic 
point is that the availability of time series data of this sort provides 
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the basis for decision-makers to detect changes in critical student 
behaviors and to formulate and test various explanations for such 
changes. In this way^ changes in institutional policy or program can 
have a firmer empirical basis* -r 

Possible Benefits ^ 

A fully operating student-based management information system 
would have a number of advantages over more conventional adminis- 
trative tasks. Assuming that the data were disseminated to key ad- 
ministrators and to faculty at the departmental levels many members 
of the academic community could become involved in the task of 
interpreting the data and devising appropriate policies based on those 
data. One weakness of traditional attempts at institutional evaluation 
-^and I would include here the typical case study prepared for ac- 
crediting teams — is that only a limited number of peop?e within the 
academic commimity are involved in the development of the data 
base and the interpretations of results. A comprehensive student*based 
system, on the other hand, could theoretically involve all members of 
the academic community in the complex task of interpretation. 

A closely related virtue of the student-based approach is that the 
dialogue ^,conceming institutional policy would be diverted from a 
preoccupation \Ath budgets and resources to a concern with the 
educational development of students. Discussions about who should 
get what piece of the financial pie -are inherently competitive and 
conflict-ridden. -There is only so much of the pie to go aroimd and 
giving more to one group implicitly deprives other groups. Focusing 
on the educational development of students, howeverv is a positive 
process where all interelsted parties presumably share common ob- 
jectives. Improving the educational effectiveness of any given unit 
does not limit or inhibit the ability of any other unit to do the same. 
This is not to say that a student-based system obviates the need for 
budgetary debates, but rather that final decisions about budgetary 
allocations would presumably focus more on their intended educa- 
tional outcomes. 

A student-based system also has the potential to benefit institu- 
^tions economically, the most obvious area being enrollments. It seems 
likely that a^ properly run student-based system could eventually help 
to reduce attrition rates. Also, there is the possibility that a heavily 
student-oriented institution might eventually be able to attract mere 
applicants: the information network for prospective college students 
is such that the word eventually gets around about institutions that 
have particularly interesting or unique programs. 

Accreditation's RoIq 

^What role, if any, can the accreditation process play in encour- 
aging institutions to tak^ a more student-oriented approach to plan- 
ning and decision-making? I beheve that accreditation can play a 
significant role because accrediting teams normally communicate 
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directly with the top administration of the institution. The adminis- 
tration, of course, determines the type of information system used in 
decision-making. Perhaps the best device for encouraging administra- 
tors to take a more student-oriented approach is the institutional 
self -study. A well-done self -study can reveal a great deal about the 
institution's communication and information system. A major com- 
plaint heard about self-studies is that they are generally viewed as 
something an institution does for the benefit of an accrediting team, 
rather than as a regular part of the institution's information- 
gathering activity. It is frustrating to encounter an institutional men- 
tality which fails to see the need for an ongoing self-analysis as an 
essential part of the management process. 

Kow can the accrediting organizations begin to encourage institu- 
tions to modify their information-gathering and self-assessment 
activities in the directions I have suggested? One very direct approach 
is in the types of questions asked of the institution: 

□ What kind of information concerning student involvement or 
lack of involvement is currently available to key institutional 
personnel? ^ 

D Do classroom instiructors and faculty advisors regularly have 
access to data on students' academic involvement and academic 
progress? 

D Do student .personnel administrators periodically receive in- 
formation on students' extracurricular pairticipation and social life? 

□ Do appropriate data get disseminated in a comprehensible form 
and in time, to take appropriate action? 

n Does the top administration take the initiative in surveying 
faculty and the staff needs for information on student progress and 
development? 

□ Is the administration responsive to' requests for better 
information? ' ■ - " 

D What does the administration know about the quality of in- 
struction, as perceived, by students and various departments? 

^ □ What do students think of the advising they receive in various 
departments? 

□ How adequate are different types of student services? 

Simply asking such questions is one means of making institutions 
more aware of the need for this type of information and of its poten- 
tial value in planning and decision-making. 

It also might be possible io develop a new set of criteria or speci- 
fications for the institutional self-study. In this regard, I think it 
w;ould be important for the accrediting teams to encourage institu- 
tions to begin to think of the self -study as an ongoing activity rather 
than as so^mething done every so often for . the benefit of an accred- 
iting team. 

The student-based system represents a kind of continuing institu- 
tional evaluation. Although I agree with those evaluation enthusiasts 



who argue that the very process of evaluation can be useful — that it 
is good for the soul to engage in critical self-evaluation — I am con- 
cerned that too many of us have accepted the idea^that evaluation is 
something you do when you have a problem that needs resolution. 
My impression is that the nature of institutional politics is such that 
once a problem has been identified and positions have been taken, no 
amount of objective data collection and self -study is going to budge 
people from their preconceived positions. Therefore, I am pessimistic 
about the usefulness of a student-based management system which is 
designed primarily to provide data to solve institutional conflicts. On 
the other hand, the student*based system can be extremely useful in 
identifying problems before they have come to the attention of the 
academic community in general and before political positions have 
been taken. One might refer to this- as a kind of **gratuitous** evalua- 
tion in which potential institutional problems are identified throu^ 
tiie **early waming system." 

Postscript 

My view that institutions need to take a more student-oriented 
approach to administration and planning implies a concept of "qual- 
ity*' that deviates considerably from our traditional definitions. Thus, 
a "high quality** institution under this new conception is one' that 
knows what's happening to its students. FjLirther, the high quality 
institution has a student information-gathering and disseminating 
mechanism which enables it to make appropriate adjustments in pro- 
grams or policies when the student data indicate that change or 
improvement is needed* In other words, quality is equated here not 
with physical facilities or curricula but rather with a. continuing 
process of critical self-examination that focuses oh the institution's 
contribution to the student's intellectual and personal development. 
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GOALS, outcomes; and 

ACADEMIC EVALUATION 



Howard R. Bowen 



The jai^on of higher education is everchanging. Among the new 
and fashionable words are evaluation, consumer protectian, and 
accountability. The language may be new, but the ideas expressed by 
these words have long been the basic concepts underlying accredita- 
tion. Accreditation, however, may be entering a new era. As higher 
education has expanded and proliferated, the need and the demand 
of society for consumer protection and accountability have become 
more urgent. In meeting new societal needs and demands, the pro- 
cedures of accreditation must become more concemer^ with outcomes 
and less preoccupied with resource inputs. Accrediting bodies will 
have to provide some of the leadership, the encouragement, and even - 
the clout necessary to persuade institutions to be more mindful of 
their own outcomes. 

Kenneth Boulding, at a conference this past spring at the Univer- 
sity of Michigan, said: 

Hie demand for greater accoimtability, it seems to me, 
must be met in two ways. On the one hand, imiversities must 
not simply shrug it off as meaningless. They should devote 
resources to improving the feedback from the processes in 
which they engage. The imiversity should conceive of itself as 
an experiment in teaching, learning,' and the expansion of 
knowledge; Its information collection apparatus should be 
designed with useful feedback in view. This is by no means 
easy. It should be regarded as a challenge worthy of the best 
intellectual resources of the institution. The problem should 
not be pushed off onto administrative officers or even the 
specialized research institute, though there is^much to be §aid 
for this. And the more both students and faculty and admin- 
istrators caii see themselves as participating in a continuing 
process of improvement, the more likely is improvement to 
■ take placf^ - 

Along with this, however, there must also be a constant 
campaign of explanation directed toward the public at large 
and to its representatives as to the necessity for freedom. 
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leeway, redundancy, space to move, opportunities for experi- 
ment, and so on. ^thout this, the drive for efficiency can be 
disastrous.^ 

The second part of Dr. Boulding's prescription seems to be of 
special relevance to voluntary accreditation, the very essense of which 
is to maintain the freedom of institutions while keeping them socially 
responsible. 

Efficiency^ Accountability and Management 

In contemporary discussions of academic affairs, three of the 
most widely discussed concepts are efficiency^ accountability^ and 
management. As in the case of the weather, everyone talks about 
efficiency, accountability, and management but nobody does much 
about them — at least in a fundamental way. The reason is that all 
three of these concepts are utterly meaningless in the absence of valid 
assessments of. outcomes, and no one quite knows how to estimate 
outcomes except through vague intuitive procedures. 

Efficiency, by definition, is expressed as a ratio between out- 
comes and resources employed — that is, between benefits and costs, 
or less elegantly, between outputs and inputs. Similarly, accountabil- 
ity implies an obligation to produce outcomes tiiat are commensurate 
with the costs and to demonstrate that the outcomes justify the costs. 
Likewise, management implies that different technologies (or methods 
or procedures) will produce different results or outcomes, and that it is 
the task of managers to select those technologies which will produce 
the best outcomes in relation to the amount of resources employed* 
In each case, the assessment of outcomes need not be precise and 
need no^: be expressed in cardinal numbers, but to make any one of 
the thxee concepts meaningful there must be some notion of out- 
comes expressed in terms of "more or less." All decisions about staff, 
curricula, teaching methods, facilities, institutional organization, 
governance, and even finance can be arrived at rationally and delib-^ 
erately only if there is some knowledge of effects on outcomes. 

It is a fact that cherished values are not readily suseptible to pre- 
cise measurement. Friendship, love, beauty,' honor, patriotism, and 
fairness are assessed by means^ of an art that is based upon intuitive 
judgment.' When such intuitive judgment is formalized and institution- 
alized, it is called criticism. Our powers of criticism are developed in 
proportion to our sehsitivity, and critical judgments usually are based 
upon standards derived from traditioii. They flow out of human ex- 
perience. The influential critic is a person who has keen sensitivity 
coupled with knowledge of traditional standards and who is thus able 
to reach judgments that gain wide acceptance, i.e., the art critic, the 
social critic, and the educational critic. Even in the area of economics, 
where money serves as a measure of value, the critic has an important 
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role. For example, an economy that maximizes monetary values may 
be criticized because it does not produce a fair distribution of income, 
or because it does not give sufficient attention to the environment, 
or because it ignores job satisfaction, or because it produces personal 
alienation, or be<?aus*» it encourages vulgar taste. 

In the world as it is— even in the economy— there is no way to 
solve questions of value by easy quantitative formulas. There is no 
way to side-step intuitive judgment and criticism, with all the pitfalls 
they entail. This is especially so in education, wi.ere the purpose is to 
facilitate the sound development of tmioue human beings and the 
ultimate end is the gooa J4fe. It would be foolish to suppose that 
educational outcomes could be readily quantified and the processes 
of judgment and criticism could Bfe^ wholly dispensed with. In educa- 
tional planning and decision-making, we ^tre forced to practice the 
art of criticism nased on intuitive judgments. As Cardinal Newman 
observed : 

A university is. ..an Alma Mater, knowing her children one by 
one, not a foundry, or a mint, or a treadmill.^ 
Similarly, John Dewey wrote: 

What the best and wisest parent wants for a child, that must 
the community want for all it« children. Any ether ideal for 
our schools is narrow and unlovely; acted upon it destroys 
our democracy 

And an economist friend of mine, Sar Levitan, summed it all up 
when he said, "Statistics are no substitute for judgment.*** 

Yet the concept of efficiency' has a place in all human endeavors, 
even in such inscruitable areas as foreign policy, military strategy, 
religion, art, marriage, poker. Education is surely no exception; there 
are better ways and poorer ways of going about teaching-learning, and 
there are also more expensive and less expensive ways of going about 
It. The most efficient ways are those which yield the highest ratio of 
results to cost. Just as families or churches or nations can be more or 
less efficient in achieving their benevolent purposes, colleges and 
universities can also be more or less efficient. And colleges and uni- 
versities could be more efficient if they paid greater attention to 
discovering their outcomes. 

The Link Between Outcomes^ and Management 

At present, institutions know very little about their results and 
next to nothing about the effects of changes in their procedures and 

2. John H. f Cardinal) Newman, The Scope and Nature of University Education, 
(New York: E. P. Dutton, 1958), p. 122. 

3. John Dewey, 0« Education, (R. D. Archambault, editor), (Chicago: Univer- 
sity of Chicago Press, 1974), p. 295. ' 
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4- From a poster disp&yed in Dr. Levi tan's office. 
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methods on the results. Even much of what passes for evaluation has 
little ccmtact with true outcomes— ^;hat is, with what happens to 
students, whose development is the object of the whole hi^er 
educational enterprise. There have been sporadic one-time studies of 
' outcomes in particular institutions and also a number of one-time 
studies of small samples of institutions, but there have been few 
systematic ongoing efforts to assess outcomes, and certainly few 
cases where the study of outcomes has been linked with management. 
Such studies are urgently needed if institutions are to have better 
information for management and accountability, and if the profession 
is to leam more about the consequences of alternative procedures 
and methods. Without such knowledge, institutions are destined 
merely to follow tradition, or to do what is expedient in the li^t of 
prevailing pressures of the market and of politics, or to be vulnerable 
to every fad that sweeps through the educational community, or to 
manage by intuition, or to do some of all four. 

One may respect tradition because it is the result of a kind of 
natural selection achieved through many trials and errors; one may 
understand the need for adjusting to political and market pressures; 
one may believe that institutions should be open to the new; and one 
may respect intuition especially when it is practiced by experienced 
and sensitive persons. Yet, decision-making need riot be based solely 
on tradition,, political pressure, fad, and intuition when these can be 
supplemented by knowledge. If the nature of our task is- such that we 
often must fly blind, it would be good to have radar to give at least 
a general sense of direction. We all have seen the faculty meeting in 
•which learned men and women debate educational or curricular 
policy as embodied in degree requirements. They make extended 
sp>eeches on the advantages of this or that requirement without the 
remotest notion of what difference their proposals would make in 
the development of the students for whom they are prescribing. 

Seven Basic Principles 

I suggest the following basic principles that should be followed 
in the identification and evaluation of the outcomes of particular 
• colleges or universities, such principles confined to the outcomes of 
undergraduate education, but equally applicable, I believe, to ad- 
vanced professibnad and graduate study, research and scholarship, and 
public service. 

I. The study of outcotnes should avoid the common confusion of 
inputs and outputs. Most evaluation of institutions is conducted in 
terms of such variables as faculty-student ratios, number of PhDs on 
the faculty, size of endowment, current expenditures per student, 
SAT- scores, number of library books, range of facilities and equip- 
ment, value of physical plant, extracurricular program?, etc. These 
factors are all inputs and it is by no means established that there is 
any systematic and positive correlation between these inputs and the 
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true outcomes defined in terms of the |>ersonal development of 
students. 

In view of the wide differences in expenditures per student am ong 
seemingly comparable institutions, the burden of proof is ui>on the 
high-cost institutions to demonstrate that their generous expenditures 
actually y ;ld commensurate outcomes. It is not even appropriate to 
define outcomes in terms of the experience of students in connection 
with their college education. Such measures as attendance in the 
libraries and laboratories, number and typ>e of courses taken, attend- 
ance at public events, etc., are technologies, not results. Even such 
variables as absenteeism or dropout rates are not conclusive indicators 
of outcomes. It is possible that students are doing the right thing, in 
terms of their personal development, by staying away from formal 
programs, or by changing institutions, or by dropping out of college 
altogether. The only valid tests of outcomes are: Whet happens 'in the 
development of persons? How do persons change and grow as a result 
of their college experiences? 

II- A second principle is that assessment should be linked to all the 
major goals of education, and not be confined just to aspects of 
human development that can be easily measured or that are related 
to economic success. In general, five kinds of gosds will be of interest 
to most institutions. They are: 

n Cognitive development of students, including verbal and quan- 
titative skills, substantive, knowledge, rationality, critical thinking, 
intellectual tolerance, lifelong learning. 

D' Aesthetic sensibility. 

□ Emotional and moral development, including personal self- 
discovery, human understanding, religious interest, psychological 
well-being. 

D Practical competence relating to citizenship, economic pro- 
ductivity,' family life, consumption, leisure, and health. 

□ Direct satisfactions and enjoyment from colltege education 
during the college years and in later life. 

The goals of an institution are its intended outcomes, and hence 
institutions are prone to think of their outcomes in terms of their 
intentions and hopes. But ^hey should be on the lookout also for 
imin tended outcomes. Some of these may by serendipity be positive, 
but some may be negative. For example, college may generate dis- 
couragement, boredom, suppression of creativeness, acquisition of 
bad habits, needless failure, and even suicide. And the^e negative 
outcomes should be taken into account. 

III. A third principle is that educational outcomes should relate to the 
development- of whole persons. Most evaluation of students employs 
instruments that measure only particular aspects of their personalities. 
Such instruments may measure verbal skills, religious interest. 
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aesthetic sensibility, general knowledge, or life goals. Elach measure- 
ment says something about groups of students with respect to a 
particular dimension 'of their f>ersonaiitie5, but teUs very little about 
what happens to individual students considered as whole persons. The 
average changes on these various personality dimeraaons usually are 
small and lead the unwary to conclude that college really does not 
matter very much. Contrarily, collie may exert a major impact on 
students, considered as whole human beings, even when Jliere is little 
or no change in average scores on particular characteristics. 

The explanation of this seeming paradox is that there can be 
enormous dispersion among individuals with respect to particular 
characteristics while the averages change little or not at all. Every 
tabulation of scores on specialized test instruments shows wide dif- 
ferences in what happ>ens to individuals as they are affected by col- 
lege. On any given dimension some students will gain ground, some 
will lose ground, and some will not change at all. Aggregates and 
averages reflect the central tendency of all these wide variations and 
conceal the substantial differences exF>erienced by different individ- 
uals. For example, Feldman and Newcomb (1973, p. 9) cite a study 
in which a group of fireshmen achieved an average score on religious 
value of 41.75 and the identical students as seniors scored 41.57. 
But the individual students in the group showed changes of -19, -16, 
-14, -12, -2, +1, +3, +7, +11, +12, +16. A comparison of the two 
averages would lead to the conclusion that no change had occurred; 
yet the individual scores revealed ^erormous variance. 

The purpose of higher education is not necessarily to produce 
positive changes for all its students along every dimension of person- 
ality development. ^This would clearly be impossible and undesirable. 
Colleges accept individual differences among their students and 
encourage their students to develop individually along lines consistent 
with their unique interests and talents. For any individual,* such 
development inevitably means substantial progress along some lines, 
no change along others, and regression along still others. Regression 
along some lines is not necessarily a sign of failure. For example, a 
student who in high school had devoted much time to music may 
later discover interests in science. As a result, the student may regress 
during college on aestheticism and progress on intellectuality. A stu- 
dent may come to college with highly developed mathematical skills, 
but there discover new interests in social issues that result in a 
regression in^mathematical skills but a progression in political aware- 
ness. These regressions are not necessarily negative outcomes; they 
may represent constructive changes in personality configuration as a 
result of exposure to new opportunities. 

It is unlikely that any student will progress simultaneously along 
all possible lines of development. People differ in the distribution of 
their talents; some individuals have highly specialized talents and 
interests, other more diversified ones. Few if any are so well-rounded 
that they can be expected to make equal progress across all possible 
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lines of personality development. The cultivation of any one int^est 
takes time and energy. To pursue every interest equally would be at 

- the sacrifice of significant achievement in any one area. The totally 
well-rounded individual ujho pursued every conceivable interest would 
surely be a bland, innocuous, and mediocre person, ExceQence of 
personality involves choices based on a weighing of values in relation 
to interests and talents, not the equal pursuit of every conceived 
value. There may be ^me values, such as honor or consideration for 
others, that should have high priority for all; but among the wide 
range of values that life presents, there are tradeoffs involving choices 
based on the unique characteristics of each individual. This being so, 
each individual does and should make use of the opportunities avail- 
able in the college environment for the development of his or her 
uniqu** personality. In the process most individuals may change 
substantially, but the averages for each personality characteristic may 
change relatively little. 

To study the true outcomes of higher education in all their 
variability, then, calls for the study of individual students in some 
depth. One must ask, not what was the average gain in verbal skills of 
a college's entire student body, but rather what happ>ened in the total 
personality and in the life experience of individual students as a result 
of their college careers when these students are seen as whole human 
beings. The few studies that focus on whole' persons yield results that 
come closer to the changes that educators observe in their students 

during and after college, than typical studies based on average changes 
in particular dimensions of personality. 

I V . A fourth principle is that outcomes assessment should be based 
upon the study of alumni as well as of students. Such study raises 
questions that educ^stional research has barely touched. There is 
almost no knowledge Qf carryover to adult life of substantive leamir<T- 
The residue of a coUege education— after the initial foi^ettL-.^ <Vi 
detail— is a virtual mystery. Moreover, we should be interested in ti:** 
values and attitudes of^alumni, their interests, their citizenship, their 
family life, and their careers as these may have been affec^ted by their • 
college experiences. 

V. A fifth principle is that the outcomes assessment should be con- 
cerned with change in students as a result of their college experiences, 
not merely with their absolute level of performance during and after 
college. By far the most important factor determining the perform- 
ance of students during and after college is their background and 
ability at the time or admission to college- If an institution admits 
mainly bright young people from affluent families, the performance 
of Its students in their studies and in their later careers will almost 
certainly be superior to those of students of an institution that admits 
many students of limited backgrounds and abilities. For the more 
selective coUeges and universities- to, take credit for the achievements 
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of their alumni, as they often do, borders at best on the naive and at 
worst on the fraudulent. There is evidence to indicate that many 
non -selective institutions bring about greater changes m their students 
than some selective ones, and it is change not absolute perfoxmance 
that is the criterion of true educational outcomes. The change may be 
measured relative to change achieved by those who do not attend 
college, *or relative to change achieved by students of other institu- 
tions, or relative change achieved by the subject institution in past ' . 
years. ' ^ 

VI. A sixth principle is that the evaluation scheme must be prac- 
ticable, not too time-consuming or expensive. It should concentrate 
on major goals and avoid trivial detail, and should be based on 
small sample^t of students and alumni and not try to cover the entire 
imiverse. The experiences associated with evaluation should be de- 
sighed to have educational benefits to students of suffic lent value to 
justify the cost on direct educational grounds. Faculty and student 
time involved in outcomes assessment should be incorporated^ in the 
regular educational program and not require additional expenditures. 
The test of practicability means also that the results must be. under- 
standable to the public and their representatives, and must carry con- 
viction. Otherwise, outcomes assessment will be worse than useless in 
achieving accountability- Practicability implies also that the study of 
outcomes should be linked with serious educational decision-making 
and should not be used merely for public relations. In short, practica- 
bility calls for simplicity, low cost, and integrity. 

VII. The seventh principle is that outcomes assessment should develop 
from the bottom up within individual colleges and universities rather 
than be imposed from the top down by federal or state government 
or by national accrediting bodies. There is need for a great deal of 
experimentation in concepts and methods. Assessment should be 
related to the special missions or philosophies of particular institu- 
tions. It should not be restricted to procedures and instruments that 
can be used nationally. Institutions should not have to adjust their 
goals and programs to the requirements of a standardized evaluation 
scheme or to measure their performance using uniform standards set 
by others. Clearly, there is a place for procedures and instruments 
that have wide applicability, but there is great danger that outcomes 
evaluation will be dominated by conventional academic testing. This, 
in my opinion, would be a grave mistake. The true outcomes of edu- ■ 
cation cannot be expressed — except in small part— through techniques 
-that produce neat and tidy numerical scores. They call for serious 
exploration of what happens in the lives of people as a result of their 
college experiences. Sound outcomes assessment will focus on stu- 
dents as p>ersons, and will provide a vehicle by which institutions can 
become acquainted with their students as living persons rather than 
as numbers in a computer or grades on* transcripts. ^ 
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The'Practical Issues of Oi^tcomes Assessment 

There are several practical issues to be considered in successful 
outcomes measurement. 

First, there is the technical problem of controlling for extraneous 
variables. How does one know whether what happens to students in 
college is due to native ability, family background, the mass media, 
or to the normal maturation process? This problem can be solved in 
two ways. One way is to record change over time in the results ob- 
tained by the institution. If over time, one observes that gains be- 
tween *Jie freshmen and senior years are increasing or "decreasing, 
that b'jcomcs evidence that the institution is progressing or retro- 
gressir g. An evaluation program carried on year after year establishes 
its ov n norms and provides a way of measuring, or at least judging, 
insticutional progress. Another is to join with other comparable insti- 
tutions in the evaluative process, so that each institution could com- 
pare Its results with others. Ideally, a group of cooperating institu- 
tions would be composed of those which are not mutually competitive 
so that tliere would be no diffidence about sharing information. 

A second problem pertains to publicity of results from outcome 
studies. Should they be held in confidence or made public? (My 
preference would be to make the evaluative process public.) The 
entiie college or university community ought to be psychologically 
mvolved in evaluation. The evaluation ought to be conceived as a 
concerted effort by everyone in the institution— students, faculty, 
administration, and other staff— to bring about educational improve^ 
ment. In case of poor performance or retrogression or sluggish im- 
provement, the whole community should know and be concerned 
and aU should lend their efforts to correct the problem. Good public 
relations would flow from an honest and competent effort to evalu- 
ate results, whereas a policy of publicizing good nt ws and suppressing 
bad news would be the worst kind of public relations. Just as most 
mstitutions publish their financial and enrollment data, they might 
publish their educational results as w^|. The results might at times be 
humblmg and sobering; but those institutions which could manage 
and plan on the basis of clear objectives and measured performance 
would outdistance those which fly blind. 

Finally is the question of cost. The program outlined in this paper 
would not be easy or cheap. An institution ev^ntu?»ily might devote 
as much as one to three percent of its educational budget to evalua- 
tion and related research and development. Substantial parts of the 
program could be started soon, and without exorbitant cost, but the 
entuc concept would require a great deal of invention, learning, 
op>ercitional time, and money. By using sampling techniques, in the 
begim-img with very small samples, the cost could be held down. 

In *;he .'startup phase, an institution could secure considerable 
help from foundations or special gifts, but not in the long pull. I 
beheve the program should be incorporated into the teaching load of 
the faculty by giving students credit for participation in the program 



and thus reducing the ordinary teaching load. Students might earn 
the equivalent of credit fbr at least one course if they took the tests, 
kept the records, attended the interviews, and fill, d out the question- 
naires which would be part of the program. The experience of parti- 
cipation in an effort to achieve solid evaluation would have educa- 
tional significance justifying credit. By offering credit, faculty time 
could be diverted from ordinary teaching to the evaluation program. 

Institutions that will give leadership in the matter of evaluation 
will find the cos»t^ returned to them many times over, partly because 
evaluation is the way to demonstrable quality and partly because it is 
the way to true accountability. Money, for particular institutions and 
also for higher education as a whole, will follow demonstrated quality 
and accountability. 

A Cpncfuding Note 

I do not pretend that outcomes assessment as I conceive it would 
be easy. I am not even sure it is feasible. Yet I think it is uigent that 
educators try to learn more about their outcomes. And I believe that 
through such efforts carried on independently in many institutions, 
we might gradually through trial and error develop sound methods 
which would supplement our intuitive judgments and serve as a cor- 
rective for the wishful thinking and the empty rhetoric in which we 
all indulge. Over time, the assessment process might contribute to 
efficiency and might demonstrate the value of higher education in 
ways that would strengthen public confidence in it. 
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WHAT HAVE WE LEARNED? 

A Summary of the 1978 COPA Summer Conference 

Charles M. Chambers 

«T the substantive and informative papers of Alexander 

W. Astm and Howard R. Bowen, it is hard to imagine a conference 
^mg bold enough to try to accomplish more. Yet scheduled between 
these two presentations were a number obsessions, panels, and work- 
shops which delved further into the conference theme of "Evaluating 
Educational Quality." Many of these sessions emphasized the appl^ 
cation of- new evaluative concepts to accreditation and, in a number 
of msten^s, reinforced many of the same points made by Dr. Astin 
and Dr. Bowen. This synthesis will not recoimt individual session 
topics of particular speeches. It will go beyond merely reporting the 
co^iinon Ideas and themes which flowed through the conference, and 
will attempt to state what was learned at the conference and how this 
knowledge might be applied to the work of accrediting agencies. 

OveraU, the conference affirmed that the measurement of out- 
comes not only is desirable but is essential if postsecondary education 
is to gam a better understanding of quality factors in institutions and 
programs. Indeed, outcome approaches go far beyond accreditation's 
focus on the evaluation of educational quality and are viewed by 
many as central to the planning, design, and operation of soimd edu- 
^tional activities. Thi^s, by measuring outcomes, accreditation would 
become an mtegrated part of the entire educational process ra£her 
thair^ ad hoc diagnostic tool brought in from outside the institution 
rae symmetry which outcomes assessment offers both to the 
establishment and to the evaluation of educational activities woiUd 
^atly enhance the efficiency, consistency, and reliability of accred- 
itation As a bonus, if it. were based on outcomes, accreditation 
would directly reflect the accomplishments of the educational process 
and would be heavily relied on in reporting and accounting to various 
.constituencies and users. 

The Outcomes Assumption 

A working assumption in assessing outcomes is that the educa- 
tional proce^ can be structured along lines which make prominent 
the goals to be achieved. Before accreditation attempts to make use 
of such an approach to evaluation, we would want to be sure that the 
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assumption on which it is based is solid. The study directed by 
Norman Bums* presented a test and verification of this hypothesis 
and demonstrated that institutions should have no qualms about 
developing and applying new policies and procedures. In particular, 
the study demonstrated the value to institutions of establishing clear 
objectives for their educational services. These objectives should be 
stated in explicit terms which are amenable to external evaluation. 
Also, in conjunctipn with the educational objectives, the institution 
should establish an ongoing system of self-assessment which can 
analyze, describe, and document when and how the objectives are 
being achieved. Finally, the whole process of establishing objectives 
and assessing achievement should be organized so that an effective 
validation by peer experts can be accomplished by the accrediting 
agencies. 

Despite the desirability of moving in this direction, it remains a 
fact that evaluation and validation are amorphous processes which 
compound and confound dimensions of quality in education. Thu^, 
accreditation must try to learn from others who are designing and 
creating academic programs using outcomes methods. Much progress 
has been made by a number of independent researchers- Collectively, 
they have produced -an assortment of resource materials including 
directories, taxonomies, and case studies. These all point to signifi- 
cant uses of outcome approaches by educators. It is an opportune 
time for accreditation to formulate and adopt appropriate new 
evaluation processes. The situation faced today by accreditation is- 
somewhat akin to the early days of instructional technology. Both 
the mechanical equipment and the subject matter experts were avail- 
able, but the learning theory specialists still were needed to adapt the 
program material to the technical format. Accreditation faces a 
similar problem of having to take much that was presented at the 
conference and build it into working procedures. 

The Quantitative Franle of Reference 

One important observation made during the conference was that 
whenever a development in accreditation is proposed, there is always 
a certain amount of enthusiasm from some quarters that accredita- 
tion must become fairer, more rigorous, and more accountable. It is 
suggested that there be mort> objective approaches to collecting 
statistical data which, when weighted with the proper coefficients, 
will yield a numerical measure of accreditable quality. Such a'quanti- 
tative approach, however, is devoid of, the flexible assessment of 
quality which accreditation now offers. At .best, accreditation must 
remain a mix of both the quantitative and qualitative aspects of 
education. 



1. Norman Bums, liirector^ Evaluation of Institutions of Postsecondary Accred- 
itation: Assessment in Terms of Outcomes Through Institutional Self-Study. 
(Washington: Council on Postsecondary Accreditation, 197i8). 
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Of course, there is a role for analytic studies. Quantitative factozs 
can help set the frames of reference in which the subjective expertise 
of the peer educator is brought to bear. This in no sense computerizes 
accreditation, and we still must arrive at that end point of classical, 
rhetorical criticism where people who know their stuff say "having 
considered all this, that, or the other, I like it or I don't like it 
because...". 

. A Rood illustration of the importance of well defined frames o^ 
reference to accreditation is given by the surveys occasionally made 
of various schools and departments and which purport to rank them 
by reputation. Here, peers are asked to examine a list of schools or 
programs and "using their expertise" rank them from the best on 
down. A simple tally of the votes^is then used to see which is "num- 
ber one." Such surveys are not given much deference, because the 
peers' expertise is squandered by lack of a proper evaluative frame- 
work based on current knowledge (a self-study), concurrent examina- 
tion (a team visit), and coordinated decision-making (a commission 
review). ^ \ 

Evaluation via Outcomes , . 

While there are a variety of types of outcome measures, those 
which can reasonably ^be used for evaluating educational quality 
should meet the following two tests. 

First, each outcome should represent something which is teach- 
able; that is, it is generally accepted that the particular educational 
objective adopted can be achieved through an instructional or didac- 
tic process. This is not to say that there are not other appropriate 
objectives— fox example in research, student services, finance, and 
community relations— which an institution should set for itself. 
Rather, the latter outcomes must be viewed as leading to the educa- 
tional objectives which, much remain foremost in any assessment of 
educational quality. 

Second, each outcome should measure something judged relevant 
to the graduate's functioning throughout adult life. This is not to 
suggest that only things which have a practical use should be taught. 
Rather, it permits educational worth to be judged on the broad base 
of the student's entire life rather than on limited terms of what is 
useful and marketable at the time of graduation. The scope of this 
test obviously is quite extensive. Both the specific types of technical 
competencies derived from professional programs as well as the 
quality of life and responsible citizenship potential of the libera] arts 
programs can be represented in appropriate outcome terms. In keep- 
ing with the role of accreditation r*ot co overly intrude into the in- 
ternal academic affairs of. the institui^'ii or program, this test merely 
serves to screen out the seductive sophistry and rambling relevance of 
the innovative and attempts to make explicit that which is not only 
worthwhile, but also worthy of evaluation. 

One example of outcomes which meet these tests is given by the 
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suggestions of the COPA Task Force on Educational Quality. At the 
baccalaureate level the institution is expected to provide communica- 
tion skills, problem-solving skills, decision-making skills, and value- 
clarifying skills. These outcomes are the essence of an adult function- 
ing in a complex society and can be applied to programs in the arts 
and humanities, science and technology, and the social studies. 

On a broader, institutional- plane a second example is found in the 
characteristics of excellence in hi^er education which form the basis 
for accreditation by the Commission on Higher Education of the 
Middle States Association of Colleges and Schools. Among other 
. things, these characteristics si:>eak of programs or courses which de- 
velop abilities to form independent judgment, to weigh values, and to 
understand fundamental theories. Further, where appropriate, offer- 
ings are attuned to professional or occupational requirements. The 
characteristics also encourage clearly stated purposes and objectives 
coupled with 'a F»ersistent concern as to the relation between objec- 
tives and outcomes. Finally, each institution is expected to create an 
atmosphere conducive to developing the students* education beyond, 
the minimum degree requirements. 

More generally, outcomes can be identif icKi at various levels within 
the institution. For example^, an institution-wide outcome would be 
to maintain its reputation for academic quality by continuing to earn 
one or more types of accreditation. Outcomes at the program level 
could be providing a general education, offering preparation for entry 
into an occupation, or developing the qualiHcations necessaury for> 
acceptance into graduate- school. At the course level, the primary 
outcome is to teach the subject niattr- defined by the syllabus. Re- 
lated outcomes might be to prepare ' le student for a more advanced 
course, or to r:iabie the student to integrate separate fundamental 
V courses studied earlier. " 

Self- Study and Team Review 

In order for outcomes approaches to succeed, it was felt by 
conferees that such approaches must have the support and involve- 
■ ment of the faculty actually offering the projnrams to be evaluated. 
This . is an ideal match to the philosophy of the accreditation self- 
study. It was observed that as a tool ^or formulating and focusing 
faculty judgment the self-study remains the central genius of 
accreditation. 

Current research findings were presented at the conference com- 
paring studies made of outside, monitored internal, and strictly 
internal evaluation reports. These findings show that internal evalua- 
tion is as reliable and accurate a method of depicting the current 
situation as the other two. This does much to dispel the criticism that 
the self-study — on its face — is self serving, biased and unreliable. 

Although the self-study remains an int^ral part of the process, a 
number of suggestions were made about the types of questions the 
self-study should try to answer. These questions may be> generally 
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characterized as open-ended, with a strong emphasis on student in- 
volvement in the educational process. Typical ones include: 

O What data does the administration provide on student activities 
Mrhich is of use to and is used by the factdty? 

□ What are the students* attitudes about the advising system? 

P Do advisors and other faculty have access to information on 
student participation in extra-curricular activities? 

Thus, a starting point in making this a part of -accreditation is to 
review the self-study handbooks to see where changes can be made 
which encourage an institution to look at itself from a student in- 
volvement,*outcomes-oriented perspective. 

The emphasis on student activities stems from research that 
strongly suggests a cause and effect link between student involvement 
and subsequent learning that occurs. It was not made clear, however, 
whether self-selection or native ability is involved, i.e., "good students 
achieve good performance;" but, regardless, from accreditation's 
point of view, the relationship between student involvements and 
quahty outcomes are clear enough to warrant increased self-study in 
student activity areas. 

Because of the open-ended nature of this approach, an important 
caveat is in x>rder. In assessing outcomes, accreditatic r -^iist exercise 
much care in distinguishirig between "public data*' and * riv ite data." 
On the one hand, public data is prepared by the institu' t fo^ general 
dissemination and repotting. It is usually too sumir i -u.ed .uid too 
targeted to an external audience to be useful in e' A^at i'-n On the 
other hand, private data is that which rcan be used . r valuation and 
can be managed within the confidentiality of the r .reiitai .oii proc- 
ess. For example, a student assessment of individi fa- Uty members 
to be published in a campus-wide directory wou- i^^^ useful to 
accreditation than similar evaluations of faculty i t students 
were asked to respond department-by-departmeni . znr -h sharper 
. questions about educational quality. 

Once the self-study is completed to the satisfaction of the ac- 
crediting body, it should continue to be validated by a visiting team 
of peer experts. Because the self-study will have changed in format 
ard content, the members of the team must be specially trained to 
apply their educational expertise in a framework built around out- 
come measures. This involves more than just orienting the team to 
how the institution applied outcome approaches in its self-study 
Team members must receive specific formal training in the methodol- 
ogy of outcomes evaluation so that their report can mesh with the 
self-study. 

The Use of Accreditation 

In a different vein, the conference explored the dimensions of 
self-regulation and the extent to which an institution should peri- 
odically examine itself, with or without accreditation. While this is 
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nt>i uirectly related to outcomes assessment, there was consensus that 
the accreditation process should not be viewed as the end of a study 
of past events, but rather the beginning of plans for future activities 
designed to bring about changes indicated by the accreditation review. 
Furthermore, if accrediting agencies are to give increased attention to 
outcome approaches, then accreditation can help insure that the 
institutions get the benefit of what is an expensive and time-consum- 
ing exercise. Put bluntly, accreditation should strive vigorously to 
overcome the **thank goodness it's over for another eight years'* 
syndrome found on so many campuses. 

Schematically, a strategy designed to encourage an institution to 
use its own self-study for future planning might proceed along these 
five steps: 

C After the institution has sought accreditation and completed 
the self-study phase, tne campus committee develops a concluding 
chapter in which it identifies the problems that exist and presents an 
action plan for solving them. As a minimum, the plan would be 
certain to fulfill the accrediting agencies' criteria, but ideally it would 
bring about even greater improvements on the campus. 

O After reviewing the self-study and its final chapter, the ac- 
crediting agency would be asked to suggest resource persons who 
would complement the individual educational objectives of the insti- 
tution and have expertise in the typyes of solutions proposed for its 
problem areas. These persons could later possibly serve as the visiting 
team or committee. 

□ At this point, the resource persons receive and review the self- 
study and offer advice to both parties about proceeding with the 
plan. Their report is not a formal accreditation finding and if the plan 
presented in the last chapter of the self-study is found wanting, it is 
returned to the institution with suggestions for Turther amplification 
and refinement. 

a After nine to twelve months of using tne plan, the institution 
is assigned a formal team which then visits the school anci writes a 
report judging how well the school is solving those problems which 
could prevent it from achieving its stated objectives. 

□ The team report and the self-study are then examined by the 
Commission in reaching a final accreditation decision. 

Such a process would result in the following somewhat futuristic 
definition of accreditation: 

An institution or program is accredited if it: (1) can dempn- 
strate that it can accurately assess its own potential relative 
to the evaluative criteria of the accrediting agency; (2) that it 
can use this information to design and implement a plan to 
achieve its potential; and (3) that the accrediting agency can 
validate this plan and its success through peer review. 
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It was proposed that one to three percent of an institution's 
budget be officially allocated to assessment, evaluation, and planning. 

Validity 

One remaining theme from the conference involved the nagging 
doubt about the validity of outcomes assessments. Unfortunately, 
the concept of formal, statistical validity of subjective evaluative 
methods such as accreditation remains as elusive^ as ever. Until more 
studies and analyses have been completed we must rely on the good 
offices of the expert critic working voluntarily through the checks 
and balances of the accrediting process to evaluate educational 
quality. 

Finally, a bonus of the conterence which reinforced the impor- 
tance of outcomes assessments was the report on the COP A project 
on the evaluation of nontraditional education.^ Because nontradi- 
tional education by definition does not fit traditional patterns, we 
learned that it must be looked at directly in terms of its educational 
accomplishment. The researchers in this project concluded that non- 
traditional education is most naturally evaluated and accredited, in 
terms of outcomes, an important lesson, if not a new theme. 



2. Grover J. Andrews, Assessing Nontraditional Education, four vols., (Wash- 
ington: Council on Postsecondary Accreditation, 1978). 
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